Papers with domain-specific language models
Knowledge-enhanced Response Generation in Dialogue Systems: Current Advancements and Emerging Horizons (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | Knowledge-enhanced Dialogue Systems (KEDS) are a new approach to enhancing human-machine interaction through natural language. |
| Approach: | This tutorial provides an in-depth exploration of Knowledge-enhanced Dialogue Systems (KEDS) it aims to elucidate their significance, highlight advances made using deep learning, and pinpoint the current challenges. |
| Outcome: | The tutorial aims to give attendees a comprehensive understanding of KEDS, and highlight advances made using deep learning and pinpoint the current challenges. |
Evaluating Pretraining Strategies for Clinical BERT Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing generic language models in specialized domains may be sub-optimal due to domain differences. |
| Approach: | They propose various strategies for adapting a generic language model to the target domain and various forms of vocabulary modifications to fine-tune it. |
| Outcome: | The proposed strategies outperform a general-domain language model but little difference in performance between the models. |
BUSTER: a “BUSiness Transaction Entity Recognition” dataset (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Natural Language Processing has seen major breakthroughs in the last few years, but transferring these advances into industry applications can be difficult. |
| Approach: | They propose to use a BUSiness Transaction Entity Recognition dataset to support industry-oriented research by exploiting both general-purpose and domain-specific language models. |
| Outcome: | The proposed model is the best performing model and an additional silver corpus to BUSTER. |
MedDistant19: Towards an Accurate Benchmark for Broad-Coverage Biomedical Relation Extraction (2022.coling-1)
Copied to clipboard
| Challenge: | Relation extraction in the biomedical domain is challenging due to the lack of labeled data and high annotation costs. |
| Approach: | They propose to use distant supervision to pair knowledge graph relationships with raw texts to tackle the scarcity of annotated data and to validate their results. |
| Outcome: | The proposed benchmarks are more accurate and consistent with existing benchmarks and show that there is no train-test leakage. |
EconBERTa: Towards Robust Extraction of Named Entities in Economics (2023.findings-emnlp)
Copied to clipboard
Karim Lasri, Pedro Vitor Quinta de Castro, Mona Schirmer, Luis Eduardo San Martin, Linxi Wang, Tomáš Dulka, Haaya Naushan, John Pougué-Biyong, Arianna Legovini, Samuel Fraiberger
| Challenge: | Adapting general-purpose language models to specific domains has proven to be effective in tackling downstream tasks such as impact evaluation. |
| Approach: | They propose to use EconBERTa, a large language model pretrained on scientific publications in economics, and ECON-IE, based on an expert-annotated dataset of economics abstracts for Named Entity Recognition (NER). |
| Outcome: | The proposed model outperforms EconBERTa on the downstream NER task and ECON-IE on the economics abstracts. |
VE-KD: Vocabulary-Expansion Knowledge-Distillation for Training Smaller Domain-Specific Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | VE-KD is a method that balances knowledge distillation and vocabulary expansion with the aim of training efficient domain-specific language models. |
| Approach: | They propose a method that balances knowledge distillation and vocabulary expansion with the aim of training efficient domain-specific language models. |
| Outcome: | VE-KD outperforms DistilBERT and Adapt-and-Distill in biomedical domain tasks . compared with other methods, it outperformed Distilbert and adapted-and distill . |
PPORTAL_ner: An Annotated Corpus of Portuguese Literary Entities (2024.lrec-main)
Copied to clipboard
| Challenge: | Annotated corpus of 25 literary texts provides a rich set of annotations for Named Entity Recognition models. |
| Approach: | They propose an annotation dataset that simplifies the development of Named Entity Recognition models for Portuguese literary texts. |
| Outcome: | The proposed dataset simplifies the development of Named Entity Recognition models for Portuguese literary works. |